Apparatus and method for inferencing topology of multiple cameras network by tracking movement

ABSTRACT

Provided are an apparatus and a method for tracking movements of objects to infer a topology of a network of multiple cameras. The apparatus infers the topology of the network formed of the multiple cameras that sequentially obtain images and includes an object extractor, a haunting data generator, and a haunting database (DB), and a topology inferrer. The object extractor extracts at least one from each of the obtained images, for the multiple cameras. The haunting data generator generates appearing cameras and appearing times at which the moving objects appear, and disappearing cameras and disappearing times at which the moving objects disappear, for the multiple cameras. The haunting DB stores the appearing cameras and appearing times and the disappearing cameras and disappearing times of the moving object, for the multiple cameras. The topology inferrer infers the topology of the network using the appearing cameras and appearing times and the disappearing cameras and disappearing times of moving objects. Therefore, the apparatus accurately infers topologies and distances among the multiple cameras in the network of the multiple cameras using the cameras and appearing and disappearing times at which the moving objects appear and disappear. As a result, the apparatus accurately track the moving objects in the network.

TECHNICAL FIELD

The present invention relates to a network of multiple cameras, and moreparticularly, to an apparatus and a method for tracking movements ofobjects to infer a topology of a network of multiple cameras.

BACKGROUND ART

Accidents and crimes have been increased with the complexity anddiversity of modern society. Thus, safety and surveillance have beenincreasingly demanded. In order to satisfy the demand of safety andsurveillance, unmanned monitoring systems, etc. have been developed tomonitor principal facilities and keep public security in crime-riddendistricts.

As a practical and economic alternative, such unmanned monitoringsystems have been developed into intelligent monitoring systems, whichdetect and track moving objects, beyond the development of Digital VideoRecorders (DVRs). Such intelligent monitoring system technology is todetect and distinguish moving objects from image sequences. In order torealize such intelligent monitoring system technology, there is requiredtechnology for automatically distinguishing and tracking objects in theenvironments of multiple monitoring cameras.

In the early algorithms for tracking objects, several cameras trackobjects using camera calibration and overlapping Fields of View (FOV).As one of such algorithms, there is a method of constituting camerasusing a compensated camera to overlap viewers with one another and thencalculating handovers of tracked objects.

Another method is to track objects in cameras of which viewers do notoverlap with one another. In other words, in this method, the viewers donot overlap with one another in the cameras in order to match transitiontimes of the objects with shapes of the objects. However, in thismethod, the transition times of the objects must be used, and movingmethods of people must be pre-informed.

There is another method of describing motion patterns of objects fromtwo cameras using a stochastic transition matrix. This method requireslearned data in order to confirm coherences among the cameras.

Besides the above-described method, there is a method of constituting are-entry period and then using objects that are observed within a giventime. In this method, the re-entry period is expressed as a histogramusing the observed objects, and links among cameras are detected basedon the histogram.

There is another method of modeling posterior probability distributionsand color variations of space-time links among cameras using progressivelearning. This method is to verify coherences of objects in multiplecameras that do not overlap, based on a progressive learning process forsimilarities among colors. Here, the similarities among the colors aredetermined using a Consensus-Color Conversion of Munsell (CCCM) colorspace. Links among the multiple cameras are determined after thedeterminations of the similarities among the colors.

In the above-method, entries and re-entries of objects are calculated asconditional transition probabilities within a given time to determinethe links among the multiple cameras. Here, in order to furtheraccurately determine the links, blocks within screens of the multiplecameras are divided into maximally small sizes to remove unnecessarylinks and sort out effective links. After the links are sorted out, theposterior probability distributions are used to further accuratelydetermine the coherences of objects that are observed among links.

Another method is to confirm coherences of objects using relationsbetween FOV lines in the environment in which camera viewers do notoverlap with one another in real-time situations. In this method, theFOV lines of the cameras, which do not overlap with one another, extendto form virtual lines, and then minimum distances from the virtual linesto estimated positions of objects are calculated in order to determinewhether the objects are the same objects.

In the above-described object tracking methods, cameras are constitutedusing various methods. Here, the cameras may be classified into singlecameras and multiple cameras. Also, the multiple cameras may beclassified into overlapping multiple cameras and non-overlappingmultiple cameras according to whether images overlap with one another.The overlapping multiple cameras overlap camera images, while thenon-overlapping multiple cameras do not overlap camera images.

The single cameras are mainly used to perform defined functions ofspecific spatial areas like over-speeding car regulating cameras. Themultiple cameras are used to keep and monitor safety in indoor andoutdoor wide areas. The overlapping multiple cameras are used to allowtwo or more cameras to share a predetermined area so as to minutelyanalyze images or used to produce 3-dimensional (3-D) images throughstereo vision technology.

However, the non-overlapping multiple cameras are mainly used inmonitoring systems for wide areas due to installation and maintenancecost. External identifiers of objects and haunting time intervals of theobjects must be considered to track the objects in the non-overlappingmultiple cameras.

DETAILED DESCRIPTION OF THE INVENTION Technical Problem

The present invention provides an apparatus and method for trackingmovements of objects to infer a topology of a network of multiplecameras.

Technical Solution

According to an aspect of the present invention, there is provided anapparatus for tracking movements of objects to infer a topology of anetwork of multiple cameras sequentially obtaining images, including anobject extractor, a haunting data generator, a haunting database (DB),and a topology inferrer. The object extractor may extract at least onemoving object from each of the obtained images, for the multiplecameras. The haunting data generator may generate appearing cameras andappearing times at which the moving objects appear, and disappearingcameras and disappearing times at which the moving objects disappear,for the multiple cameras. The haunting DB may store the appearingcameras and appearing times and the disappearing cameras anddisappearing times of the moving objects, for the multiple cameras. Thetopology inferrer may infer the topology of the network using theappearing cameras and appearing times and the disappearing cameras anddisappearing times of the moving objects.

The object extractor may include a moving object extractor and acharacteristic extractor. The moving object extractor may removebackground areas from the obtained images to obtain foreground areas soas to extract the foreground areas as the moving objects. Thecharacteristic extractor may extract characteristics of the movingobjects.

The haunting data generator may determine cameras on which the movingobjects appear, as the appearing cameras and times at which the movingobjects appear, as the appearing times if the moving objects appear onthe cameras, and if the moving objects disappear from the cameras,determine the cameras from which the moving objects disappear, as thedisappearing cameras and times at which the moving objects disappear, asthe disappearing times.

According to another aspect of the present invention, there is provideda method of tracking movements of objects to infer a topology of anetwork of multiple cameras, including: sequentially obtaining imagesfrom the multiple cameras; extracting at least one moving object fromeach of the obtained images; determining and registering appearingcameras and appearing times at which the moving objects appear, anddisappearing cameras and disappearing times at which the moving objectsdisappear, for the multiple cameras; and inferring the topology of thenetwork using the appearing cameras and appearing times and thedisappearing cameras and disappearing times.

The extraction of the moving objects may include removing backgroundareas from the obtained images to obtain foreground areas so as toextract the foreground areas as the moving objects.

The determination and registration of the appearing cameras andappearing times and the disappearing cameras and disappearing times mayinclude: if the moving objects appear on the cameras, determining thecameras on which the moving objects appear, as the appearing cameras andtimes at which the moving objects appear, as the appearing times; and ifthe moving objects disappear from the cameras, determining the camerasfrom which the moving objects disappear, as the disappearing cameras andtimes at which the moving objects disappear, as the disappearing times.

The inference of the topology may include: if the moving objects appearon the cameras, determining whether the moving objects are new movingobjects; if the moving objects are the new moving objects, registeringthe moving objects and generating image nodes of the appearing cameras,and if the moving objects are registered moving objects, comparingmoving times of the moving objects with a threshold moving time; and ifthe moving times are greater than the threshold moving time,re-registering the moving objects as new moving objects and generatingimage nodes of the appearing cameras, and if the moving times are notgreater than the threshold moving time, comparing the appearing cameraswith previous disappearing cameras to determine the topology of thenetwork.

The extraction of the moving objects may include extractingcharacteristics of the moving objects, and the determination of whetherthe moving objects are the new moving objects may include comparing thecharacteristics of the moving objects with characteristics of previouslyextracted moving objects to determine whether the moving objects are thenew moving objects.

The determination of the topology of the network may include: if theprevious disappearing cameras are not registered or the appearingcameras of the moving objects are not the same as the previousdisappearing cameras, determining whether the moving objects disappearfrom other cameras to generate image nodes or overlapping nodes; and ifthe appearing cameras of the moving objects are the same as the previousdisappearing cameras, generating virtual nodes of the appearing cameras.

The generation of the virtual nodes may include generating invisibleedges to the virtual nodes, wherein weights of the invisible edges tothe virtual nodes are the moving times.

The generation of the image nodes or the overlapping nodes may include:if the moving objects do not appear on the other cameras, generating theimage nodes of the appearing cameras; and if the moving objects appearon the other cameras, generating overlapping nodes between the appearingcameras and the other cameras.

The generation of the image nodes may include generating invisible edgesto the image nodes, and the generation of the overlapping nodes mayinclude generating visible edges to the overlapping nodes, whereinweights of the invisible edges to the image nodes and weights of thevisible edges to the overlapping nodes are the moving times.

The method may further include: if the moving objects disappear from thecameras, determining whether the moving objects appear on other camerasexcept the disappearing cameras; if the moving objects do not appear onthe other cameras, generating non-overlapping nodes; and if the movingobjects appear on the other cameras, determining whether previousappearing cameras are the same as the disappearing cameras to generatecamera nodes or regression visible edges.

The generation of the camera nodes or the regression visible edges mayinclude: if the previous appearing cameras are not the same as thedisappearing cameras, generating image nodes of the disappearingcameras; and if the previous appearing cameras are the same as thedisappearing cameras, generating regression visible edges that regressto the disappearing cameras.

The generation of the image nodes of the disappearing cameras mayinclude generating overlapping edges to the image nodes, wherein weightsof the overlapping edges to the image nodes and weights at theregression visible edges are the moving times.

ADVANTAGEOUS EFFECTS

In an apparatus for inferring a topology of a network of multiplecameras according to an embodiment of the present invention, topologiesand distances among the multiple cameras may be accurately measured inthe network of the multiple cameras using appearing and disappearingcameras and appearing and disappearing times at which moving objectsappear and disappear. Thus, the moving objects may be accurately trackedin the network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an apparatus for tracking movements ofobjects to infer a topology of a network of multiple cameras accordingto an embodiment of the present invention;

FIG. 2 is a flowchart of a method of tracking movements of objects toinfer a topology of a network of multiple cameras according to anembodiment of the present invention;

FIG. 3 is a flowchart of operation 250 of the method of FIG. 2;

FIG. 4 is a flowchart of a part 270_1 of operation 270 of the method ofFIG. 2;

FIG. 5 is a flowchart of an other part 270_2 of operation 270 of themethod of FIG. 2;

FIG. 6 schematically illustrates a network space of multiple cameras inwhich a topology inferring method of an embodiment of the presentinvention is experimented;

FIG. 7 illustrates data experimented in the network space of FIG. 6,according to an embodiment of the present invention;

FIG. 8 illustrates data experimented in the network space of FIG. 6,according to another embodiment of the present invention; and

FIG. 9 illustrates topology relations among multiple cameras that areinferred using the data of FIGS. 7 and 8.

BEST MODE FOR CARRYING OUT THE INVENTION

The attached drawings for illustrating preferred embodiments of thepresent invention are referred to in order to gain a sufficientunderstanding of the present invention, the merits thereof, and theobjectives accomplished by the implementation of the present invention.

Hereinafter, the present invention will be described in detail byexplaining preferred embodiments of the invention with reference to theattached drawings. Like reference numerals in the drawings denote likeelements.

FIG. 1 is a block diagram of an apparatus for tracking movements ofobjects to infer a topology of a network of multiple cameras accordingto an embodiment of the present invention. Referring to FIG. 1, atopology inferring apparatus 100 according to the present embodimentinfers a topology of a network 115 that includes multiple cameras 110_1through 110_N that sequentially obtain images. The topology inferringapparatus 100 includes an object extractor 130, a haunting datagenerator 150, a haunting database (DB) 170, and a topology inferrer190.

The object extractor 130 extracts at least one movement from an imagethat is obtained by each of the multiple cameras 110_1 through 110_N.For convenience, it is described in the present embodiment that theobject extractor 130 extracts movements respectively from the multiplecameras 1101 through 110_N. While this is only an example, it will beunderstood by those of ordinary skill in the art that the objectextractor 130 may be realized in various forms.

The object extractor 130 includes moving object extractors 131_1 through131_N and characteristic extractors 133_1 through 133_N. The movingobject extractors 131_1 through 131_N remove background areas from theimages that are obtained by the multiple cameras 110_1 through 110_N toobtain foreground areas and then extract the foreground areas as movingobjects. The characteristic extractors 133_1 through 133_N extractcharacteristics of the moving objects.

The haunting data generator 150 generates appearing camera and appearingtimes at which moving objects appear, and disappearing cameras anddisappearing times at which the moving objects disappear, for themultiple cameras 110_1 through 110_N.

Moving objects haunt, i.e., appear on and/or disappear from, cameraswith the movements of the moving objects. In this case, the hauntingdata generator 150 may detect when the moving objects haunt (appear onand/or disappear from) which cameras (appearing cameras or disappearingcameras) to generate haunting data (the appearing cameras and appearingtimes, and the disappearing cameras and disappearing times).

In more detail, if the moving objects appear cameras, the haunting datagenerator 150 may determine cameras on which the moving objects appear,as appearing cameras, and times at which the moving objects appear, asappearing times, in order to generate haunting data. If the movingobjects disappear from the cameras, the haunting data generator 150 maydetermine cameras from which the moving objects disappear, asdisappearing cameras, and times at which the moving objects disappear,as disappearing times, in order to generate haunting data.

The haunting DB 170 stores the appearing cameras and appearing times andthe disappearing cameras and disappearing times of the moving objectsrespectively for the multiple cameras 110_1 and 110_N. It will beunderstood by those of ordinary skill in the art that haunting data maybe stored using various methods besides the above-described method.

The topology inferrer 190 infers the topology of the network 115 usingthe appearing cameras and times and the disappearing cameras and timesof the moving objects. In the present embodiment, the topology inferrer190 compares haunting data of a currently appearing or disappearingmoving object with previously stored haunting data of the currentlyappearing or disappearing moving object whenever moving objects appearor disappear, in order to infer the topology. An operation of thetopology inferring apparatus 100, in particular, an operation of thetopology inferrer 190, will now be described in detail with reference toFIGS. 1 through 5.

FIG. 2 is a flowchart of a method of tracking movements of objects toinfer a topology of a network of multiple cameras according to anembodiment of the present invention. In operation S210, the multiplecameras 110_1 through 110_N respectively sequentially obtain images inorder to detect haunts of moving objects.

In operation S230, the object extractor 130 extracts at least one movingobject from each of the images that are obtained by the multiple cameras110_1 through 110_N. Here, the object extractor 130 may further extractcharacteristics of the moving objects. In other words, as describedabove, the moving object extractors 131_1 through 131_N removebackground areas from the obtained images to obtain foreground areas andextract the foreground areas as the moving objects.

In the present embodiment, the moving object extractors 131_1 through131_N may use a robust segmentation that complexly uses color, edge,movement, and disparity information, in order to extract the foregroundareas.

In other words, the moving object extractors 131_1 through 131_Ncalculate smooth disparities of the obtained images, compare colors,edges, movements, and disparities, and separate the moving objects fromstatic backgrounds. Also, when it is assumed that the disparities of themoving objects are smoothly changed within limited disparity ranges,i.e., limited depth ranges, to separate only the moving objects, themoving object extractors 131_1 through 131_N track moving objectsincluding depth information among the separated moving objects in orderto extract the moving objects.

The characteristic extractors 133_1 through 133_N extractcharacteristics of the moving objects that are extracted using theabove-described method. In the present embodiment, the characteristicextractors 133_1 through 133_N may extract the characteristics of themoving objects using a grid patch classification method. If the gridpatch classification method is used, the characteristic extractors 133_1through 133_N split appearances of blobbed moving objects (e.g.,persons) into respective parts in the foreground areas. In the presentembodiment, if a moving object is a person, the characteristicextractors 133_1 through 133_N divide the moving object into a head, abody, hands, legs, etc. in order to use a hair color, whether the personwears a hat, clothes colors, etc. as identifiers of the moving object.

As will be described later, in operation S270, the topology inferrer 190to determine coherences of the moving objects when inferring thetopology of the network 115 of the multiple cameras. Here,characteristics of the moving objects, e.g., hair colors, whether themoving objects wear hats, clothes colors, etc., may be used asidentifiers of the moving objects.

Referring to FIG. 2 again, in operation S250, the haunting datagenerator 150 determines and registers appearing cameras and appearingtimes at which the moving objects appear, and disappearing cameras anddisappearing times at which the moving objects disappear, for themultiple cameras.

FIG. 3 is a flowchart of operation S250 of the method of FIG. 2. If themoving objects appear on cameras as described above, the haunting datagenerator 150 determines the cameras on which the moving objects appear,as appearing cameras, and times at which the moving objects appears asappearing times, in order to generate haunting data in operation S251.If the moving objects disappear from the cameras, the haunting datagenerator 150 determines the cameras from which the moving objectsdisappear, as disappearing cameras, and times at which the movingobjects disappear, as disappearing times, in order to generate hauntingdata in operation S253. The generated haunting data, i.e., the appearingcameras and appearing times and the disappearing cameras anddisappearing times, are stored in the haunting DB 170.

As previously described, in operation S270, the topology inferrer 190infers the topology of the network 115 using the appearing cameras andappearing times and the disappearing cameras and disappearing times. Thetopology inferrer 190 compares haunting data of appearing ordisappearing moving objects with previously stored haunting datawhenever the moving objects appear or disappear, in order to infer thetopology. The case whether the moving objects appear will be describedseparately from the case whether the moving objects disappear in orderto describe the operation of the topology inferrer 190 with reference toFIGS. 4 and 5.

FIG. 4 is a flowchart of a part 270_1 of operation 270 of the method ofFIG. 2, i.e., a flowchart of a case whether moving objects appear oncameras. If the moving objects appear on the cameras, appearing camerasand appearing times of the moving objects obtained in theabove-described operations are registered.

Referring to FIG. 4, in operation S271, the topology inferrer 190determines whether a moving object currently appearing on cameras is anew moving object. As previously described, the object extractor 130 mayextract moving objects along with characteristics of the moving objects.In the present embodiment, the characteristic extractors 133_1 through133_N may extract characteristics of moving objects, e.g., hair colors,clothes colors, body sides, etc. In this case, the topology inferrer 190may compare characteristics of the current moving object withcharacteristics of a previous moving object to determine whether thecurrent moving object is a new moving object.

If it is determined in operation S 271 that the currently appearingmoving object is the new moving object, the topology inferrer 190registers the currently appearing moving object in operation S272. Inoperation S273, the topology inferrer 190 generates image nodes of thecameras on which the current moving object appears, i.e., image nodes ofappearing cameras corresponding to the current moving object.

If it is determined in operation S271 that the current moving object isnot the new moving object, i.e., the current moving object is apreviously registered moving object, the topology inferrer 190 comparesa moving time of the current moving object with a threshold moving time.Here, the moving time refers to a time interval between neighboringevents of consecutive events (i.e., appearing or disappearing events)occurring in a moving object. Thus, if a moving object appears, a movingtime corresponds to a difference between an appearing time of the movingobject and a time at which the moving object disappears or appears justbefore appearing.

Moving objects may move within the network 115. Alternatively, themoving objects may leave from the network 115 and then enter the network115. The latter case corresponds to a situation that is not necessaryfor inferring a topology, and such a situation is required to beexcluded. In the present embodiment, if time intervals among eventsconsecutively occurring in moving objects are within the thresholdmoving time, the topology inferrer 190 determines that the movingobjects have moved within the network 115.

Therefore, if the moving times are greater than the threshold movingtime, the topology inferrer 190 determines that the moving objects haveleft from the network 115 and then entered the network 115. In otherwords, the topology inferrer 190 re-registers the current moving objectas the new moving object in operation S272 and then generates imagenodes of the appearing cameras in operation S273.

If the moving times are not greater than the threshold moving time, thetopology inferrer 190 compares the appearing cameras with cameras fromwhich the current moving object disappeared previously, i.e., previousdisappearing cameras, in operation S275 in order to determine thetopology of the network 115. An operation of determining a topology of anetwork by comparing previous disappearing cameras will now bedescribed.

If the appearing cameras are the same as the previous disappearingcameras, the topology inferrer 190 generates virtual nodes of theappearing cameras in operation S276 and generates invisible edges to thevirtual nodes in operation S277. The topology inferrer 190 registers themoving times as weights of the invisible edges to the virtual nodes inoperation S283.

If the previous disappearing cameras are not registered or the appearingcameras are not the same as the previous disappearing cameras, thetopology inferrer 190 determines whether the moving objects haveappeared on other cameras, in operation S278 in order to generate imagenodes or overlapping nodes.

In more detail, if the moving objects do not appear on the othercameras, the topology inferrer 190 generates image nodes of theappearing cameras in operation S281 and generates invisible edges to theimage nodes in operation S282. The topology inferrer 190 registers themoving times as weights of the invisible edges to the image nodes inoperation S283.

If the moving objects appear on the other cameras, the topology inferrer190 generates overlapping nodes between the appearing cameras and theother cameras in operation S279 and generates visible edges to theoverlapping nodes in operation S280. In operation S283, the topologyinferrer 190 registers the moving times as weights of the visible edgesto the overlapping nodes.

It has been described the operation of the topology inferrer 190 whenmoving objects appear on cameras. An operation of the topology inferrer190 when the moving objects disappear from the cameras will now bedescribed with reference to FIG. 5.

FIG. 5 is a flowchart of other part 270_2 of operation 270 of the methodof FIG. 2. If the moving objects disappear from the cameras, thetopology inferrer 190 determines whether the moving objects appear onother cameras except the to disappearing cameras, in operation S291.

If the moving object does not appear on the other cameras, the topologyinferrer 190 generates non-overlapping nodes in operation S292 andgenerates visible edges to the non-overlapping nodes in operation S293.If the moving objects appear on the other cameras, the topology inferrer190 determines whether previous appearing cameras are the same as thedisappearing cameras, in operation 294. If it is determined in operationS294 that the previous appearing cameras are not the same as thedisappearing cameras, the topology inferrer 190 generates camera nodesor regression visible edges in operation S295.

In more detail, if the previous appearing camera is not the same as thedisappearing camera, the topology inferrer 190 generates image nodes ofthe disappearing camera in operation S296 and generates overlappingedges to the image nodes in operation S297. Here, the topology inferrer190 registers the moving times as weights of the overlapping edges inoperation S298.

If the previous appearing cameras are the same as the disappearingcamera, the topology inferrer 190 generates regression visible edgesregressing to the disappearing cameras. Here, the topology inferrer 190registers the moving times as weights of the regression visible edges inoperation S298.

In the present embodiment, the above-described operations may berepeated on moving objects, which enter and move within the network 115of cameras, to train the moving objects so as to constitute a space-timetopology relation of the network 115. Tracking of the moving objectswithin the network 115 may be accurately performed using such a topologyrelation.

An example of constituting a space-time topology relation of the network115 in a real space will now be described with reference to FIGS. 6through 9.

FIG. 6 schematically illustrates a network space of multiple cameras onwhich a topology tracking method of an embodiment of the presentinvention is experimented. FIGS. 7 and 8 respectively illustrate dataobtained from the experiment performed in the network space of FIG. 6.FIG. 9 illustrates a topology relation of a network of multiple camerasthat is inferred using the data of FIGS. 7 and 8. The experiment in thenetwork space of FIG. 6 is performed under the following conditions.

A method of used in the experiment is realized using C++ and OpenCVunder the environments of Intel Dual Core 2.4 GHz CPU and Windows XP of1 GB Memory Spec. Also, Samsung Dome Camera SID-45 is used as cameras,and obtained images of 320×240 are processed as 15 frames per second. Inthe experiment, appearing and disappearing cameras are detected fromFOVs of cameras with movements of a user within a building in thenetwork space of FIG. 6 to analyze topology relations among the cameras.

Table below shows movements of moving objects in the network space ofFIG. 6 for 20-minute learning in time order.

Ap- Re- Dis- Movement Transition Time Camera pear Appear appear TimeTime 09:45:22:12 c1 p1 09:45:26:87 c4 p2 09:45:30:34 c2 p1 8 09:45:31:68c4 p2 5 09:45:32:56 c1 p1 10 09:45:32:81 c3 p2 1 09:45:37:41 c2 p1 709:45:37:94 c3 p2 5 09:45:38:23 c3 p1 1 09:45:39:18 c3 p1 1 09:45:40:76c4 p1 1 09:45:40:56 c5 p2 3 09:45:45:24 c4 p1 5 09:45:46:74 c3 p1 109:45:48:21 c5 p2 8 09:45:51:81 c3 p1 5 09:45:54:81 c5 p1 3 09:46:02:64c5 p1 8 09:46:03:37 c5 p3 09:46:05:15 c5 p1 −3 09:46:11:37 c5 p3 8

As shown in Table above, three moving objects p1, p2, and p3 move forabout 50 seconds in the network space of FIG. 6. The movement of themoving object p1 will now be described.

The moving object p1 appears on a first camera C1 at 09:45:22:12. Inthis case, the moving object p1 is registered in operation S272, and animage node c1 is generated in operation S273.

The moving object p1 re-appears on a second camera C2 at 09:45:30:34 butkeeps appearing on the first camera C1. In this case, an overlappingnode o1 is generated in operation S279, and a visible edge from theimage node c1 to the overlapping node o1 is generated in operation 280.About 8 seconds are required from the appearance of the moving object p1on the first camera C1 to the re-appearance of the moving object p1 onthe second camera C2. Thus, “8” is registered as a weight of the visibleedge to the overlapping node o1 in operation is S283.

The moving object p1 disappears from the first camera C1 at 09:45:32:56.However, since the moving object p1 keeps appearing on the second cameraC2, an image node c2 is generated in operation S296, an overlapping edgefrom the overlapping node o1 to the image node c2 is generated inoperation S297, and “2” is registered as a weight in operation S298. Themoving object p1 disappears from the second camera C2 at 09:45:37:41 anddoes not appear on other cameras. Thus, a non-overlapping node no1 isgenerated in operation S292, a visible edge from the image node c2 tothe non-overlapping node no1 is generated in operation S293, and “5 isregistered as a weight in operation 298.

The moving object p1 re-appears on a third camera C3 at 09:45:38:23.Thus, an image node c3 is generated in operation S281, an invisible edgeto the image node c3 is generated in operation S282, and “1” isregistered as a weight in operation S283. The moving object p1disappears from the third camera C3 at 09:45:39:18. Thus, anon-overlapping node no2 is generated in operation S292, an invisibleedge from the image node c3 to the non-overlapping node no3 is generatedin operation S293, and “1” is registered as a weight in operation S298.

The moving object p1 re-appears on a fourth camera C4 at 09:45:40:76.Thus, an image node c4 is generated in operation S281, an invisible edgeto the image node c4 is generated in operation S282, and “1” isregistered as a weight in operation S283. The moving object p1disappears from the fourth camera C4 at 09:45:45:24. Here, since adisappearing camera is the same as the fourth camera C4, a regressionvisible edge is generated in operation S295, and “5” is registered as aweight in operation S298.

The moving object p1 re-appears on the third camera C3 at 09:45:46:74. Anode and an edge related to the re-appearance have been alreadygenerated, and thus only a transition time is required to be confirmed.The moving object p1 disappears from the third camera C3 and re-appearson a fifth camera C5 at 09:45:51:81. Thus, a non-overlapping node no3and an image node c5 are generated in operations S292 and 281,respectively, visible and invisible edges are generated in operationsS293 and 282, respectively, and “4” and “3” are registered as weights inoperations S298 and 283, respectively.

The moving object p1 disappears from the fifth camera C5 at 09:46:02:64and then re-appears on the fifth camera C5 at 09:46:05:15. Here, anon-overlapping node, which is to be generated when the moving object p1disappears, is required to be generated. However, since the movingobject p1 disappears from and re-appears on the fifth camera C5, not anon-overlapping node but a virtual node v1 is generated in operationS276. In this case, a visible edge from the image node c5 to the virtualnode v1 is generated in operation S293, a regression invisible edge atthe virtual node v1 is generated in operation S277, and weights “8” and“3” are registered in operations S298 and S283, respectively.

According to the above embodiments described with reference to FIGS. 1through 5, nodes and edges may be generated and weights may beregistered according to movements of the moving objects p2 and p3 usingsimilar methods to the above-described method. Nodes and weightsdetermined as described above are expressed as a matrix and a list inFIGS. 7 and 8, respectively. The matrix among neighboring nodes for themoving object p1 is illustrated in FIG. 7, and the list of edges andweights among the neighboring nodes is illustrated in FIG. 8.

The above-described process may be repeated on moving objects that movewithin a network of multiple cameras, using the above-described method,to learn space-time topology relations among the multiple cameras withinthe network.

The learned space-time topology relations among the multiple cameras maybe expressed as a topology-weighted graph G as illustrated in FIG. 9.Referring FIG. 9, the topology-weighted graph G is formed of nodes andedges. The nodes include image nodes, overlapping nodes, non-overlappingnodes, virtual nodes, and the edges include visible edges, invisibleedges, and overlapping edges.

Links among nodes are constituted using direction information on which amoving object appears and disappears, and a difference between anappearing time and a disappearing time corresponds to a moving time atwhich the moving object moves. Thus, the moving time is registered as aweight of an edge between nodes.

An overlapping edge refers to a case where a moving object appears ontwo or more cameras. Thus, a movement direction of the moving object maybe analyzed to infer angles among cameras. In particular, in the case ofa passage in a building or a road in which vehicles move, movingpatterns and paths of people or vehicles are constant. Thus, aninference as described above may be made. As described above, aspace-time topology may be automatically constituted through learningthat is performed for a predetermined time.

As described above, a topology relation of a network of multiplecameras, including distances among the multiple cameras, distances frommoving objects to the multiple cameras, angles of the multiple cameras,etc., may be constituted using learning according to the presentinvention. Movements of the moving objects including persons may beaccurately tracked using the topology relation. In other words, themoving objects may be sequentially tracked from the multiple camerasusing external identifiers, haunting directions, time differences, etc.of the moving objects that are obtained from the topology relation.

Object tracking technology for multiple cameras realized in embodimentsof the present invention may be widely applied in a home network fieldfor maintaining safety within a security requiring building or aresidential place such an apartment complex.

The embodiments of the present invention can be written as computerprograms and can be implemented in general-use digital computers thatexecute the programs using a computer readable recording medium. Also, astructure of data used in the above embodiments can be recorded on acomputer readable recording medium through several means. Examples ofthe computer readable recording medium include magnetic storage media(e.g., ROM, floppy disks, hard disks, etc.), optical recording media(e.g., CD-ROMs, or DVDs), and storage media such as carrier waves (e.g.,transmission through the Internet).

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims.

INDUSTRIAL APPLICABILITY

An apparatus and a method for tracking movements of objects to infer atopology of a network of multiple cameras may be applied to ubiquitousindustries including home automation, etc.

1. A method of tracking movements of objects to infer a topology of anetwork of multiple cameras, comprising: sequentially obtaining imagesfrom the multiple cameras; extracting at least one moving object fromeach of the obtained images; determining and registering appearingcameras and appearing times at which the moving objects appear, anddisappearing cameras and disappearing times at which the moving objectsdisappear, for the multiple cameras; and inferring the topology of thenetwork using the appearing cameras and appearing times and thedisappearing cameras and disappearing times.
 2. The method of claim 1,wherein the extraction of the moving objects comprises removingbackground areas from the obtained images to obtain foreground areas soas to extract the foreground areas as the moving objects.
 3. The methodof claim 1, wherein the determination and registration of the appearingcameras and appearing times and the disappearing cameras anddisappearing times comprise: if the moving objects appear on thecameras, determining the cameras on which the moving objects appear, asthe appearing cameras and times at which the moving objects appear, asthe appearing times; and if the moving objects disappear from thecameras, determining the cameras from which the moving objectsdisappear, as the disappearing cameras and times at which the movingobjects disappear, as the disappearing times.
 4. The method of claim 1,wherein the inference of the topology comprises: if the moving objectsappear on the cameras, determining whether the moving objects are newmoving objects; if the moving objects are the new moving objects,registering the moving objects and generating image nodes of theappearing cameras, and if the moving objects are registered movingobjects, comparing moving times of the moving objects with a thresholdmoving time; and if the moving times are greater than the thresholdmoving time, re-registering the moving objects as new moving objects andgenerating image nodes of the appearing cameras, and if the moving timesare not greater than the threshold moving time, comparing the appearingcameras with previous disappearing cameras to determine the topology ofthe network.
 5. The method of claim 4, wherein: the extraction of themoving objects comprises extracting characteristics of the movingobjects; and the determination of whether the moving objects are the newmoving objects comprises comparing the characteristics of the movingobjects with characteristics of previously extracted moving objects todetermine whether the moving objects are the new moving objects.
 6. Themethod of claim 4, wherein the determination of the topology of thenetwork comprises: if the previous disappearing cameras are notregistered or the appearing cameras of the moving objects are not thesame as the previous disappearing cameras, determining whether themoving objects disappear from other cameras to generate image nodes oroverlapping nodes; and if the appearing cameras of the moving objectsare the same as the previous disappearing cameras, generating virtualnodes of the appearing cameras.
 7. The method of claim 6, wherein thegeneration of the virtual nodes comprises generating invisible edges tothe virtual nodes, wherein weights of the invisible edges to the virtualnodes are the moving times.
 8. The method of claim 6, wherein thegeneration of the image nodes or the overlapping nodes comprises: if themoving objects do not appear on the other cameras, generating the imagenodes of the appearing cameras; and if the moving objects appear on theother cameras, generating overlapping nodes between the appearingcameras and the other cameras.
 9. The method of claim 8, wherein: thegeneration of the image nodes comprises generating invisible edges tothe image nodes; and the generation of the overlapping nodes comprisesgenerating visible edges to the overlapping nodes, wherein weights ofthe invisible edges to the image nodes and weights of the visible edgesto the overlapping nodes are the moving times.
 10. The method of claim4, further comprising: if the moving objects disappear from the cameras,determining whether the moving objects appear on other cameras exceptthe disappearing cameras; if the moving objects do not appear on theother cameras, generating non-overlapping nodes; and if the movingobjects appear on the other cameras, determining whether previousappearing cameras are the same as the disappearing cameras to generatecamera nodes or regression visible edges.
 11. The method of claim 10,wherein the generation of the camera nodes or the regression visibleedges comprises: if the previous appearing cameras are not the same asthe disappearing cameras, generating image nodes of the disappearingcameras; and if the previous appearing cameras are the same as thedisappearing cameras, generating regression visible edges that regressto the disappearing cameras.
 12. The method of claim 11, wherein thegeneration of the image nodes of the disappearing cameras comprisesgenerating overlapping edges to the image nodes, wherein weights of theoverlapping edges to the image nodes and weights at the regressionvisible edges are the moving times.
 13. An apparatus for trackingmovements of objects to infer a topology of a network of multiplecameras sequentially obtaining images, comprising: an object extractorwhich extracts at least one moving object from each of the obtainedimages, for the multiple cameras; a haunting data generator whichgenerates appearing cameras and appearing times at which the movingobjects appear, and disappearing cameras and disappearing times at whichthe moving objects disappear, for the multiple cameras; a hauntingdatabase (DB) which stores the appearing cameras and appearing times andthe disappearing cameras and disappearing times of the moving objects,for the multiple cameras; and a topology inferrer which infers thetopology of the network using the appearing cameras and appearing timesand the disappearing cameras and disappearing times of the movingobjects.
 14. The apparatus of claim 13, wherein the object extractorcomprises: a moving object extractor which removes background areas fromthe obtained images to obtain foreground areas so as to extract theforeground areas as the moving objects; and a characteristic extractorwhich extracts characteristics of the moving objects.
 15. The apparatusof claim 13, wherein the haunting data generator determines cameras onwhich the moving objects appear, as the appearing cameras and times atwhich the moving objects appear, as the appearing times if the movingobjects appear on the cameras, and if the moving objects disappear fromthe cameras, determines the cameras from which the moving objectsdisappear, as the disappearing cameras and times at which the movingobjects disappear, as the disappearing times.